Digging for knowledge with information extraction a case study on human gene%2Ddisease associations.

Statistical Relational Learning with Formal Ontologies.
Abstract. We propose a learning approach for integrating formal knowledge into statistical inference by exploiting ontologies as a semantically rich and fully formal representation of prior knowledge. The logical constraints deduced from ontologies can be utilized to enhance and control the learning task by enforcing description logic satisfiability in a latent multi-relational graphical model. To demonstrate the feasibility of our approach we provide experiments using real world social network data in form of a SHOIN (D) ontology. The results illustrate two main practical advancements: First, entities and entity relationships can be analyzed via the latent model structure. Second, enforcing the ontological constraints guarantees that the learned model does not predict inconsistent relations. In our experiments, this leads to an improved predictive performance. 1
Hierarchical Bayesian Models for Collaborative Tagging Systems.
Abstract—Collaborative tagging systems with user generated content have become a fundamental element of websites such as Delicious, Flickr or CiteULike. By sharing common knowledge, massively linked semantic data sets are generated that provide new challenges for data mining. In this paper, we reduce the data complexity in these systems by finding meaningful topics that serve to group similar users and serve to recommend tags or resources to users. We propose a well-founded probabilistic approach that can model every aspect of a collaborative tagging system. By integrating both user information and tag information into the well-known Latent Dirichlet Allocation framework, the developed models can be used to solve a number of important information extraction and retrieval tasks. Keywords-collaborative tagging; LDA; user modeling; I.
Tutorial summary Learning with dependencies between several response variables.

Multi%2DRelational Learning with Gaussian Processes.
Due to their flexible nonparametric nature, Gaussian process models are very effective at solving hard machine learning problems. While existing Gaussian process models focus on modeling one single relation, we present a generalized GP model, named multi-relational Gaussian process model, that is able to deal with an arbitrary number of relations in a domain of interest. The proposed model is analyzed in the context of bipartite, directed, and undirected univariate relations. Experimental results on real-world datasets show that exploiting the correlations among different entity types and relations can indeed improve prediction performance. 1
A statistical relational model for trust learning.
We address the learning of trust based on past observations and context information. We argue that from the truster’s point of view trust is best expressed as one of several relations that exist between the agent to be trusted (trustee) and the state of the environment. Besides attributes expressing trustworthiness, additional relations might describe commitments made by the trustee with regard to the current situation, like: a seller offers a certain price for a specific product. We show how to implement and learn contextsensitive trust using statistical relational learning in form of the Infinite Hidden Relational Trust Model (IHRTM). The practicability and effectiveness of our approach is evaluated empirically on user-ratings gathered from eBay. Our results suggest that (i) the inherent clustering achieved in the algorithm allows the truster to characterize the structure of a trust-situation and provides meaningful trust assessments; (ii) utilizing the collaborative filtering effect associated with relational data does improve trust assessment performance; (iii) by learning faster and transferring knowledge more effectively we improve cold start performance and can cope better with dynamic behavior in open multiagent systems. The later is demonstrated with interactions recorded from a strategic two-player negotiation scenario.
Towards LarKC A Platform for Web%2DScale Reasoning.
Current Semantic Web reasoning systems do not scale to the requirements of their hottest applications, such as analyzing data from millions of mobile devices, dealing with terabytes of scientific data, and content management in enterprises with thousands of knowledge workers. In this paper, we present our plan of building the Large Knowledge Collider, a platform for massive distributed incomplete reasoning that will remove these scalability barriers. This is achieved by (i) enriching the current logic-based Semantic Web reasoning methods, (ii) employing cognitively inspired approaches and techniques, and (iii) building a distributed reasoning platform and realizing it both on a high-performance computing cluster and via ”computing at home”. In this paper, we will discuss how the technologies of LarKC would move beyond the state-of-the-art of Webscale reasoning.
Social Network Mining with Nonparametric Relational Models.
Abstract. Statistical relational learning (SRL) provides effective techniques to analyze social network data with rich collections of objects and complex networks. Infinite hidden relational models (IHRMs) introduce nonparametric mixture models into relational learning and have been successful in many relational applications. In this paper we explore the modeling and analysis of complex social networks with IHRMs for community detection, link prediction and product recommendation. In an IHRM-based social network model, each edge is associated with a random variable and the probabilistic dependencies between these random variables are specified by the model, based on the relational structure. The hidden variables, one for each object, are able to transport information such that non-local probabilistic dependencies can be obtained. The model can be used to predict entity attributes, to predict relationships between entities and it performs an interpretable cluster analysis. We demonstrate the performance of IHRMs with three social network applications. We perform community analysis on the Sampson’s monastery data and perform link analysis on the Bernard &amp; Killworth data. Finally we apply IHRMs to the MovieLens data for prediction of user preference on movies and for an analysis of user clusters and movie clusters.
Towards Machine Learning on the Semantic Web.
In this paper we explore some of the opportunities and challenges for machine learning on the Semantic Web. The Semantic Web provides standardized formats for the representation of both data and ontological background knowledge. Semantic Web standards are used to describe meta data but also have great potential as a general data format for data communication and data integration. Within a broad range of possible applications machine learning will play an increasingly important role: Machine learning solutions have been developed to support the management of ontologies, for the semi-automatic annotation of unstructured data, and to integrate semantic information into web mining. Machine learning will increasingly be employed to analyze distributed data sources described in Semantic Web formats and to support approximate Semantic Web reasoning and querying. In this paper we discuss existing and future applications of machine 1 learning on the Semantic Web with a strong focus on learning algorithms that are suitable for the relational character of the Semantic Web’s data structure. We discuss some of the particular aspects of learning that we expect will be of relevance for the Semantic Web such as scalability, missing and contradicting data, and the potential to integrate ontological background knowledge. In addition we review some of the work on the learning of ontologies and on the population of ontologies, mostly in the context of textual data. 1
Extraction of semantic biomedical relations from text using conditional random fields.

Learning Initial Trust Among Interacting Agents.
Abstract. Trust learning is a crucial aspect of information exchange, negotiation, and any other kind of social interaction among autonomous agents in open systems. But most current probabilistic models for computational trust learning lack the ability to take context into account when trying to predict future behavior of interacting agents. Moreover, they are not able to transfer knowledge gained in a specific context to a related context. Humans, by contrast, have proven to be especially skilled in perceiving traits like trustworthiness in such so-called initial trust situations. The same restriction applies to most multiagent learning problems. In complex scenarios most algorithms do not scale well to large state-spaces and need numerous interactions to learn. We argue that trust related scenarios are best represented in a system of relations to capture semantic knowledge. Following recent work on nonparametric Bayesian models we propose a flexible and context sensitive way to model and learn multidimensional trust values which is particularly well suited to establish trust among strangers without prior relationship. To evaluate our approach we extend a multiagent framework by allowing agents to break an agreed interaction outcome retrospectively. The results suggest that the inherent ability to discover clusters and relationships between clusters that are best supported by the data allows to make predictions about future behavior of agents especially when initial trust is involved.
Structure Learning with Nonparametric Decomposable Models.

Robust multi%2Dtask learning with t%2Dprocesses.
Most current multi-task learning frameworks ignore the robustness issue, which means that the presence of “outlier ” tasks may greatly reduce overall system performance. We introduce a robust framework for Bayesian multitask learning, t-processes (TP), which are a generalization of Gaussian processes (GP) for multi-task learning. TP allows the system to effectively distinguish good tasks from noisy or outlier tasks. Experiments show that TP not only improves overall system performance, but can also serve as an indicator for the “informativeness ” of different tasks. 1.
Fast Inference in Infinite Hidden Relational Models.

Tailored%2Dto%2DFit Bayesian Network Modeling of Expert Diagnostic Knowledge.

Variational Bayesian Dirichlet%2DMultinomial Allocation for Exponential Family Mixtures.
Abstract. This paper studies a Bayesian framework for density modeling with mixture of exponential family distributions. Variational Bayesian Dirichlet-Multinomial allocation (VBDMA) is introduced, which performs inference and learning efficiently using variational Bayesian methods and performs automatic model selection. The model is closely related to Dirichlet process mixture models and demonstrates similar automatic model selection in the variational Bayesian context. 1


Collaborative ordinal regression.
Ordinal regression has become an effective way of learning user preferences, but most research focuses on single regression problems. In this paper we introduce collaborative ordinal regression, where multiple ordinal regression tasks are handled simultaneously. Rather than modeling each task individually, we explore the dependency between ranking functions through a hierarchical Bayesian model and assign a common Gaussian Process (GP) prior to all individual functions. Empirical studies show that our collaborative model outperforms the individual counterpart in preference learning applications. 1.
Supervised probabilistic principal component analysis.
Principal component analysis (PCA) has been extensively applied in data mining, pattern recognition and information retrieval for unsupervised dimensionality reduction. When labels of data are available, e.g., in a classification or regression task, PCA is however not able to use this information. The problem is more interesting if only part of the input data are labeled, i.e., in a semi-supervised setting. In this paper we propose a supervised PCA model called SPPCA and a semi-supervised PCA model called S 2 PPCA, both of which are extensions of a probabilistic PCA model. The proposed models are able to incorporate the label information into the projection phase, and can naturally handle multiple outputs (i.e., in multi-task learning problems). We derive an efficient EM learning algorithm for both models, and also provide theoretical justifications of the model behaviors. SPPCA and S 2 PPCA are compared with other supervised projection methods on various learning tasks, and show not only promising performance but also good scalability.
Stochastic Relational Models for Discriminative Link Prediction.
We introduce a Gaussian process (GP) framework, stochastic relational models (SRM), for learning social, physical, and other relational phenomena where interactions between entities are observed. The key idea is to model the stochastic structure of entity relationships (i.e., links) via a tensor interaction of multiple GPs, each defined on one type of entities. These models in fact define a set of nonparametric priors on infinite dimensional tensor matrices, where each element represents a relationship between a tuple of entities. By maximizing the marginalized likelihood, information is exchanged between the participating GPs through the entire relational network, so that the dependency structure of links is messaged to the dependency of entities, reflected by the adapted GP kernels. The framework offers a discriminative approach to link prediction, namely, predicting the existences, strengths, or types of relationships based on the partially observed linkage network as well as the attributes of entities (if given). We discuss properties and variants of SRM and derive an efficient learning algorithm. Very encouraging experimental results are achieved on a toy problem and a user-movie preference link prediction task. In the end we discuss extensions of SRM to general relational learning tasks. 1
Infinite Hidden Relational Models.

Multi%2DOutput Regularized Feature Projection.

Multi%2DOutput Regularized Projection.

Hierarchy%2DRegularized Latent Semantic Indexing.

Dirichlet enhanced relational learning.
We apply nonparametric hierarchical Bayesian modelling to relational learning. In a hierarchical Bayesian approach, model parameters can be “personalized”, i.e., owned by entities or relationships, and are coupled via a common prior distribution. Flexibility is added in a nonparametric hierarchical Bayesian approach, such that the learned knowledge can be truthfully represented. We apply our approach to a medical domain where we form a nonparametric hierarchical Bayesian model for relations involving hospitals, patients, procedures and diagnosis. The experiments show that the additional flexibility in a nonparametric hierarchical Bayes approach results in a more accurate model of the dependencies between procedures and diagnosis and gives significantly improved estimates of the probabilities of future procedures. 1.
Learning Gaussian processes from multiple tasks.
We consider the problem of multi-task learning, that is, learning multiple related functions. Our approach is based on a hierarchical Bayesian framework, that exploits the equivalence between parametric linear models and nonparametric Gaussian processes (GPs). The resulting models can be learned easily via an EM-algorithm. Empirical studies on multi-label text categorization suggest that the presented models allow accurate solutions of these multi-task problems. 1.
Soft Clustering on Graphs.
We propose a simple clustering framework on graphs that encode pairwise data similarities. Unlike usual similarity-based methods, the approach softly assigns data to clusters in a probabilistic way. More importantly, a hierarchical clustering is naturally derived in this framework to gradually merge lower-level clusters into higher-level ones. A random walk analysis indicates that the algorithm exposes clustering structures in various resolutions, i.e., a higher level statistically models a longer-term diffusion on graphs and thus discovers a more global clustering structure. Finally we provide very encouraging experimental results. 1
Multi%2Dlabel informed latent semantic indexing.
Canonical Correlation Analysis (CCA) is a useful technique for modeling dependencies between two (or more) sets of variables. Building upon the recently suggested probabilistic interpretation of CCA, we propose a nonparametric, fully Bayesian framework that can automatically select the number of correlation components, and effectively capture the sparsity underlying the projections. In addition, given (partially) labeled data, our algorithm can also be used as a (semi)supervised dimensionality reduction technique, and can be applied to learn useful predictive features in the context of learning a set of related tasks. Experimental results demonstrate the efficacy of the proposed approach for both CCA as a stand-alone problem, and when applied to multi-label prediction. 1
A Probabilistic Clustering%2DProjection Model for Discrete Data.
Abstract. For discrete co-occurrence data like documents and words, calculating optimal projections and clustering are two different but related tasks. The goal of projection is to find a low-dimensional latent space for words, and clustering aims at grouping documents based on their feature representations. In general projection and clustering are studied independently, but they both represent the intrinsic structure of data and should reinforce each other. In this paper we introduce a probabilistic clustering-projection (PCP) model for discrete data, where they are both represented in a unified framework. Clustering is seen to be performed in the projected space, and projection explicitly considers clustering structure. Iterating the two operations turns out to be exactly the variational EM algorithm under Bayesian model inference, and thus is guaranteed to improve the data likelihood. The model is evaluated on two text data sets, both showing very encouraging results. 1
Multi%2Dlabel informed latent semantic indexing.
Canonical Correlation Analysis (CCA) is a useful technique for modeling dependencies between two (or more) sets of variables. Building upon the recently suggested probabilistic interpretation of CCA, we propose a nonparametric, fully Bayesian framework that can automatically select the number of correlation components, and effectively capture the sparsity underlying the projections. In addition, given (partially) labeled data, our algorithm can also be used as a (semi)supervised dimensionality reduction technique, and can be applied to learn useful predictive features in the context of learning a set of related tasks. Experimental results demonstrate the efficacy of the proposed approach for both CCA as a stand-alone problem, and when applied to multi-label prediction. 1


Dirichlet Enhanced Latent Semantic Analysis.
This paper describes nonparametric Bayesian treatments for analyzing records containing occurrences of items. The introduced model retains the strength of previous approaches that explore the latent factors of each record (e.g. topics of documents), and further uncovers the clustering structure of records, which reflects the statistical dependencies of the latent factors. The nonparametric model induced by a Dirichlet process (DP) flexibly adapts model complexity to reveal the clustering structure of the data. To avoid the problems of dealing with infinite dimensions, we further replace the DP prior by a simpler alternative, namely Dirichlet-multinomial allocation (DMA), which maintains the main modelling properties of the DP. Instead of relying on Markov chain Monte Carlo (MCMC) for inference, this paper applies efficient variational inference based on DMA. The proposed approach yields encouraging empirical results on both a toy problem and text data. The results show that the proposed algorithm uncovers not only the latent factors, but also the clustering structure. 1
Learning Gaussian Process Kernels via Hierarchical Bayes.
We present a novel method for learning with Gaussian process regression in a hierarchical Bayesian framework. In a first step, kernel matrices on a fixed set of input points are learned from data using a simple and efficient EM algorithm. This step is nonparametric, in that it does not require a parametric form of covariance function. In a second step, kernel functions are fitted to approximate the learned covariance matrix using a generalized Nyström method, which results in a complex, data driven kernel. We evaluate our approach as a recommendation engine for art images, where the proposed hierarchical Bayesian method leads to excellent prediction performance. 1
A nonparametric hierarchical bayesian framework for information filtering.
Information filtering has made considerable progress in recent years. The predominant approaches are content-based methods and collaborative methods. Researchers have largely  concentrated on either of the two approaches since a principled unifying framework is still lacking. This paper suggests that both approaches can be combined under a hierarchical Bayesian framework. Individual content-based user profiles are generated and collaboration between various user models is achieved via a common learned prior distribution. However, it turns out that a parametric distribution (e.g. Gaussian) is too restrictive to describe such a common learned prior distribution. We thus introduce a nonparametric common  prior, which is a sample generated from a Dirichlet process which assumes the role of a hyper prior. We describe e#ective means to learn this nonparametric distribution, and apply it to learn users&#039; information needs. The resultant algorithm is simple and understandable, and offers a principled solution to combine content-based filtering and collaborative filtering. Within our framework, we are now able to interpret various existing techniques from a unifying point of view. Finally we demonstrate the empirical success of the proposed information filtering methods.
Probabilistic Memory%2DBased Collaborative Filtering.
Memory-based collaborative filtering (CF) has been studied extensively in the literature and has proven to be successful in various types of personalized recommender systems. In this paper we develop a probabilistic framework for memory-based CF (PMCF). While this framework has clear links with classical memory-based CF, it allows us to find principled solutions to known problems of CF-based recommender systems. In particular, we show that a probabilistic active learning method can be used to actively query the user, thereby solving the &#034;new user problem &#034;. Furthermore, the probabilistic framework allows us to reduce the computational cost of memory-based CF by working on a carefully selected subset of user profiles, while retaining high accuracy. We report experimental results based on two real world data sets, which demonstrate that our proposed PMCF framework allows an accurate and efficient prediction of user preferences.
Generative binary codes.
The JalapeÃño Dynamic Optimizing Compiler is a key component of the JalapeÃño Virtual Machine, a new Java Virtual Machine (JVM) designed to support efficient and scalable execution of Java applications on SMP server machines. This paper describes the design of the JalapeÃño Optimizing Compiler, and the implementation results that we have obtained thus far. To the best of our knowledge, this is the first dynamic optimizing compiler for Java that is being used in a JVM with a compile-only approach to program execution.
Knowing a tree from the forest art image retrieval using a society of profiles.
This paper aims to address the problem of art image retrieval (AIR), which aims to help users find their favorite painting images. AIR is of great interests to us because of its application potentials and interesting research challenges--- the retrieval is not only based on painting contents or styles, but also heavily based on user preference profiles. This paper describes the collaborative ensemble learning, a novel statistical learning approach to this task. It at first applies probabilistic support vector machines (SVMs) to model each individual user&#039;s profile based on given examples, i.e. liked or disliked paintings. Due to the high complexity of profile modelling, the SVMs can be rather weak in predicting preferences for new paintings. To overcome this problem, we combine a society of users&#039; profiles, represented by their respective SVM models, to predict a given user&#039;s preferences for painting images. We demonstrate that the combination scheme is embedded in a Bayesian framework and retains intuitive interpretations---like-minded users are likely to share similar preferences. We report extensive empirical studies based on two experimental settings. The first one includes some controlled simulations performed on 4533 painting images. In the second setting, we report evaluations based on user preferences collected through an online web-based survey. Both experiments demonstrate that the proposed approach achieves excellent performance in terms of capturing a user&#039;s diverse preferences.
A Hybrid Relevance%2DFeedback Approach to Text Retrieval.
Relevance feedback (RF) has been an e#ective query modification  approach to improving the performance of information retrieval  (IR) by interactively asking a user whether a set of documents are relevant  or not to a given query concept. The conventional RF algorithms  either converge slowly or cost a user&#039;s additional e#orts in reading irrelevant  documents. This paper surveys several RF algorithms and introduces  a novel hybrid RF approach using a support vector machine  (HRFSVM), which actively selects the uncertain documents as well as  the most relevant ones on which to ask users for feedback. It can efficiently  rank documents in a natural way for user browsing. We conduct  experiments on Reuters-21578 dataset and track the precision as  a function of feedback iterations. Experimental results have shown that  HRFSVM significantly outperforms two other RF algorithms.
Representative Sampling for Text Classification Using Support Vector Machines.
Abstract. In order to reduce human efforts, there has been increasing interest in applying active learning for training text classifiers. This paper describes a straightforward active learning heuristic, representative sampling, which explores the clustering structure of ‘uncertain ’ documents and identifies the representative samples to query the user opinions, for the purpose of speeding up the convergence of Support Vector Machine (SVM) classifiers. Compared with other active learning algorithms, the proposed representative sampling explicitly addresses the problem of selecting more than one unlabeled documents. In an empirical study we compared representative sampling both with random sampling and with SVM active learning. The results demonstrated that representative sampling offers excellent learning performance with fewer labeled documents and thus can reduce human efforts in text classification tasks. 1
An Introduction to Nonparametric Hierarchical Bayesian Modelling with a Focus on Multi%2Dagent Learning.

GPPS A Gaussian Process Positioning System for Cellular Networks.
In this article, we present a novel approach to solving the localization  problem in cellular networks. The goal is to estimate a mobile user&#039;s  position, based on measurements of the signal strengths received from  network base stations. Our solution works by building Gaussian process  models for the distribution of signal strengths, as obtained in a series  of calibration measurements. In the localization stage, the user&#039;s position  can be estimated by maximizing the likelihood of received signal  strengths with respect to the position. We investigate the accuracy of  the proposed approach on data obtained within a large indoor cellular  network.
Collaborative Ensemble Learning Combining Collaborative and Content%2DBased Information Filtering via Hierarchical Bayes.
Collaborative filtering (CF) and contentbased filtering (CBF) have widely been used in information filtering applications, both approaches having their individual strengths and weaknesses. This paper proposes a novel probabilistic framework to unify CF and CBF, named collaborative ensemble learning. Based on content based probabilistic models for each user’s preferences (the CBF idea), it combines a society of users ’ preferences to predict an active user’s preferences (the CF idea). While retaining an intuitive explanation, the combination scheme can be interpreted as a hierarchical Bayesian approach in which a common prior distribution is learned from related experiments. It does not require a global training stage and thus can incrementally incorporate new data. We report results based on two data sets, the Reuters-21578 text data set and a data base of user opionions on art images. For both data sets, collaborative ensemble achieved excellent performance in terms of recommendation accuracy. In addition to recommendation engines, collaborative ensemble learning is applicable to problems typically solved via classical hierarchical Bayes, like multisensor fusion and multitask learning. 1
Removing redundancy and inconsistency in memory%2Dbased collaborative filtering.
The application range of memory-based collaborative filtering (CF) is limited due to CF&#039;s high memory consumption and long runtime. The approach presented in this paper removes redundant and inconsistent instances (users) from the data. Our work shows that a satisfactory accuracy can be achieved by using only a small portion of the original data set, thereby alleviating the storage and runtime cost of the CF algorithm. In our approach, we consider instance selection as the problem of selecting informative data that increase the a posteriori probability of the optimal model. We evaluate the empirical performance of our approach on two realworld data sets and attain very promising results. Data size and prediction time are significantly reduced, while the prediction accuracy is on a par with results achieved by using the complete database.
The RA Scanner Prediction of Rheumatoid Joint Inflammation Based on Laser Imaging.
We describe the RA scanner, a novel system for the examination of patients  suffering from rheumatoid arthritis. The RA scanner is based on  a novel laser-based imaging technique which is sensitive to the optical  characteristics of finger joint tissue. Based on the laser images, finger  joints are classified according to whether the inflammatory status has  improved or worsened. To perform the classification task, various linear  and kernel-based systems were implemented and their performances  were compared. Special emphasis was put on measures to reliably perform  parameter tuning and evaluation, since only a very small data set  was available. Based on the results presented in this paper, it was concluded  that the RA scanner permits a reliable classification of pathological  finger joints, thus paving the way for a further development from  prototype to product stage.
Transductive and Inductive Methods for Approximate Gaussian Process Regression.
Gaussian process regression allows a simple analytical treatment of exact Bayesian inference and has been found to provide good performance, yet scales badly with the number of training data. In this paper we compare several approaches towards scaling Gaussian processes regression to large data sets: the subset of representers method, the reduced rank approximation, online Gaussian processes, and the Bayesian committee machine. Furthermore we provide theoretical insight into some of our experimental results. We found that subset of representers methods can give good and particularly fast predictions for data sets with high and medium noise levels. On complex low noise data sets, the Bayesian committee machine achieves significantly better accuracy, yet at a higher computational cost.
Advances in Neural Information Processing Systems 13
A major problem for kernel-based predictors (such as Support Vector Machines and Gaussian processes) is that the amount of computation required to find the solution scales as O(n ), where n is the number of training examples. We show that an approximation to the eigendecomposition of the Gram matrix can be computed by the Nyström method (which is used for the numerical solution of eigenproblems). This is achieved by carrying out an eigendecomposition on a smaller system of size m &lt; n, and then expanding the results back up to n dimensions. The computational complexity of a predictor using this approximation is O(m n). We report experiments on the USPS and abalone data sets and show that we can set m n without any significant decrease in the accuracy of the solution.
Knowledge Acquisition and Automated Generation of Bayesian Networks for a Medical Dialogue and Advisory System.

Scalable Kernel Systems.
Kernel-based systems are currently very popular approaches to supervised learning. Unfortunately, the computational load for training kernel-based systems increases drastically with the number of training data points. Recently, a number of approximate methods for scaling kernel-based systems to large data sets have been introduced. In this paper we investigate the relationship between three of those approaches and compare their performances experimentally. 1
The Bayesian Committee Support Vector Machine.
Empirical evidence indicates that the training time for the support vector machine (SVM) scales to the square of the number of training data points. In this paper, we introduce the Bayesian committee support vector machine (BC-SVM) and achieve an algorithm for training the SVM which scales linearly in the number of training data points. We verify the good performance of the BC-SVM using several data sets.
Scaling Kernel%2DBased Systems to Large Data Sets.
. In the form of the support vector machine and Gaussian processes, kernel-based systems are currently very popular approaches to supervised learning. Unfortunately, the computational load for training kernel-based systems increases drastically with the size of the training data set, such that these systems are not ideal candidates for applications with large data sets. Nevertheless, research in this direction is very active. In this paper, I review some of the current approaches toward scaling kernel-based systems to large data sets.  Keywords: Kernel-based systems, support vector machine, Gaussian processes, committee machines, massive data sets  1. 
The generalized Bayesian committee machine.

Mixtures of Gaussian Processes.
We introduce the mixture of Gaussian processes (MGP) model which is useful for applications in which the optimal bandwidth of a map is input dependent. The MGP is derived from the mixture of experts model and can also be used for modeling general conditional probability densities. We discuss how Gaussian processes —in particular in form of Gaussian process classification, the support vector machine and the MGP model— can be used for quantifying the dependencies in graphical models. 1
A Bayesian Committee Machine.
The Bayesian committee machine (BCM) is a novel approach to combining estimators which were trained on different data sets. Although the BCM can be applied to the combination of any kind of estimators the main foci are Gaussian process regression and related systems such as regularization networks and smoothing splines for which the degrees of freedom increase with the number of training data. Somewhat surprisingly, we nd that the performance of the BCM improves if several test points are queried at the same time and is optimal if the number of test points is at least as large as the degrees of freedom of the estimator. The BCM also provides a new solution for online learning with potential applications to data mining. We apply the BCM to systems with fixed basis functions and discuss its relationship to Gaussian process regression. Finally, we also show how the ideas behind the BCM can be applied in a non-Bayesian setting to extend the input dependent combination of estimators.  
Robust Neural Network Regression for Offline and Online Learning.
We replace the commonly used Gaussian noise model in nonlinear  regression by a more flexible noise model based on the Student-tdistribution.
Mixture Approximations to Bayesian Networks.
Structure and parameters in a Bayesian network uniquely specify the probability distribution of the modeled domain. The locality of both structure and probabilistic information are the great benefits of Bayesian networks and require the modeler to only specify local information. On the other hand this locality of information might prevent the modeler ---and even more any other person--- from obtaining a general overview of the important relationships within the domain. The goal of the work presented in this paper is to provide an &#034;alternative&#034; view on the knowledge encoded in a Bayesian network which might sometimes be very helpful for providing insights into the underlying domain. The basic idea is to calculate a mixture approximation to the probability distribution represented by the Bayesian network. The mixture component densities can be thought of as representing typical scenarios implied by the Bayesian model, providing intuition about the basic relationships. As an additional be...
Neural%2Dnetwork models for the blood glucose metabolism of a diabetic.
We study the application of neural networks to modeling the blood glucose metabolism of a diabetic. In particular we consider recurrent neural networks and time series convolution neural networks which we compare to linear models and to nonlinear compartment models. We include a linear error model to take into account the uncertainty in the system and for handling missing blood glucose observations. Our results indicate that best performance can be achieved by the combination of the recurrent neural network and the linear error model.
Fisher Scoring and a Mixture of Modes Approach for Approximate Inference and Learning in Nonlinear State Space Models.
We present Monte-Carlo generalized EM equations for learning in nonlinear state space models. The difficulties lie in the Monte-Carlo E-step which consists of sampling from the posterior distribution of the hidden variables given the observations. The new idea presented in this paper is to generate samples from a Gaussian approximation to the true posterior from which it is easy to obtain independent samples. The parameters of the Gaussian approximation are either derived from the extended Kalman filter or the Fisher scoring algorithm. In case the posterior density is multimodal we propose to approximate the posterior by a sum of Gaussians (mixture of modes approach). We show that sampling from the approximate posterior densities obtained by the above algorithms leads to better models than using point estimates for the hidden states. In our experiment, the Fisher scoring algorithm obtained a better approximation of the posterior mode than the EKF. For a multimodal distribution, the mixture of modes approach gave superior results.
Call%2DBased Fraud Detection in Mobile Communication Networks Using a Hierarchical Regime%2DSwitching Model.
Fraud causes substantial losses to telecommunication carriers. Detection systems which automatically detect illegal use of the network can be used to alleviate the problem. Previous approaches worked on features derived from the call patterns of individual users. In this paper we present a call-based detection system based on a hierarchical regime-switching model. The detection problem is formulated as an inference problem on the regime probabilities. Inference is implemented by applying the junction tree algorithm to the underlying graphical model. The dynamics are learned from data using the EM algorithm and subsequent discriminative training. The methods are assessed using fraud data from a real mobile communication network. 1 INTRODUCTION Fraud is costly to a network carrier both in terms of lost income and wasted capacity. It has been estimated that the telecommunication industry looses approximately 2-5% of its total revenue to fraud. The true losses are expected to b...
Averaging
This paper presents Random Early Detection (RED) gate-ways for congestion avoidance in packet-switched networks. The gateway detects incipient congestion by com-puting the average queue size. The gateway could notify connections of congestion either by dropping packets ar-riving at the gateway or by setting a bit in packet headers. When the average queue size exceeds a preset threshold,the gateway drops or marks each arriving packet with a certain probability, where the exact probability is a func-tion of the average queue size. RED gateways keep the average queue size low while allowing occasional bursts of packets in the queue. During congestion, the probability that the gateway notifies a particular connection to reduce its window is roughly proportional to that connection&#039;s share of the bandwidth throughthe gateway. RED gateways are designed to accompany a transport-layer congestion control protocol such as TCP.The RED gateway has no bias against bursty traffic and avoids the global synchronization of many connectionsdecreasing their window at the same time. Simulations of a TCP/IP network are used to illustrate the performance of RED gateways. 
Nonlinear Time%2DSeries Prediction with Missing and Noisy Data.
We derive solutions for the problem of missing and noisy data in nonlinear timeseries prediction from a probabilistic point of view. We discuss different approximations to the solutions, in particular approximations which require either stochastic simulation or the substitution of a single estimate for the missing data. We show experimentally that commonly used heuristics can lead to suboptimal solutions. We show how error bars for the predictions can be derived and we show how our results can be applied to K-step prediction. We verify our solutions using two chaotic time series and the sun-spot data set. In particular, we show that for K-step prediction stochastic simulation is superior to simply iterating the predictor. 1 Introduction  Over the past years, neural networks have been applied successfully in numerous applications to nonlinear time-series prediction (Weigend and Gershenfeld, 1994). Common problems in time-series prediction are missing and noisy data. The goal is to obtai...
Combining Regularized Neural Networks.
We present our results fromcombi#EL theprediEx/WF of an ensemble of neural networks for thedi/[]#]I of hepatobiF-]F ditobiFToi /LI]# the accuracy of the diA-[Ex/W wetrai the second level networksusio the outputs of the #rst level networks asi/AL data. The second level networksachirk an accuracy thati hiat/ than that of the i/LF#-A/W networksi the #rst level. Compared to thesi/FI methodwhio averages the outputs of the #rst level networks, the second level networks are also more accurate. Wedi[AxI how the overall predi/#AL accuracy can bei/L-[][ byiAFxIF/WAx biA durix thetrai]F# of the level one networks.
A Solution for Missing Data in Recurrent Neural Networks with an Application to Blood Glucose Prediction.
We consider neural network models for stochastic nonlinear dynamical  systems where measurements of the variable of interest are only available  at irregular intervals i.e. most realizations are missing. Difculties  arise since the solutions for prediction and maximum likelihood learning  with missing data lead to complex integrals, which even for simple  cases cannot be solved analytically. In this paper we propose a speci  c combination of a nonlinear recurrent neural predictive model and  a linear error model which leads to tractable prediction and maximum  likelihood adaptation rules. In particular, the recurrent neural network  can be trained using the real-time recurrent learning rule and the linear  error model can be trained by an EM adaptation rule, implemented using  forward-backward Kalman lter equations. The model is applied to  predict the glucose/insulin metabolism of a diabetic patient where blood  glucose measurements are only available a few times a day at irregular  intervals. The new model shows considerable improvement with respect  to both recurrent neural networks trained with teacher forcing or in a free  running mode and various linear models.
Nonlinear Markov Networks for Continuous Variables.
In this paper we address the problem of learning the structure in nonlinear Markov networks with continuousvariables. Markov networks are well suited to model relationships which do not exhibit a natural causal ordering. We use neural network structures to model the quantitative relationships between variables. Using two data sets we show that interesting structures can be found using our approach. 1 Introduction  Knowledge about independence or conditional independence between variables is most helpful in &#034;understanding&#034; a domain. An intuitive representation of independencies is achieved by graphical stochastical models in which independency statements can be extracted from the structure of the graph. The two most popular types of graphical stochastical models are Bayesian networks which use a directed graph, and Markov networks which use an undirected graph. Whereas Bayesian networks are well suited to represent causal relationships, Markov networks are mostly used in cases where the...
Representing Probabilistic Rules with Networks of Gaussian Basis Functions.
There is great interest in understanding the intrinsic knowledge neural networks have acquired during training. Most work in this direction is focussed on the multi-layer perceptron architecture. The topic of this paper is networks of Gaussian basis functions which are used extensively as learning systems in neural computation. We show that networks of Gaussian basis functions can be generated from simple probabilistic rules. Also, if appropriate learning rules are used, probabilistic rules can be extracted from trained networks. We present methods for the reduction of network complexity with the goal of obtaining concise and meaningful rules. We show how prior knowledge can be refined or supplemented using data by employing either a Bayesian approach, by a weighted combination of knowledge bases, or by generating artificial training data representing the prior knowledge. We validate our approach using a standard statistical data set. 
Averaging Regularized Estimators.
We compare the performance of averaged regularized estimators.
Early Brain Damage.
Optimal Brain Damage (OBD) is a method for reducing the number of weights in a neural network. OBD estimates the increase in cost function if weights are pruned and is a valid approximation if the learning algorithm has converged into a local minimum. On the other hand it is often desirable to terminate the learning process before a local minimum is reached (early stopping). In this paper we show that OBD estimates the increase in cost function incorrectly if the network is not in a local minimum. We also show how OBD can be extended such that it can be used in connection with early stopping. We call this new approach Early Brain Damage, EBD. EBD also allows to revive already pruned weights. We demonstrate the improvements achieved by EBD using three publicly available data sets. 1 
Discovering Structure in Continuous Variables Using Bayesian Networks.
We study Bayesian networks for continuous variables using nonlinear conditional density estimators. We demonstrate that useful structures can be extracted from a data set in a self-organized way and we present sampling techniques for belief update based on Markov blanket conditional density models. 1 Introduction  One of the strongest types of information that can be learned about an unknown process is the discovery of dependencies and ---even more important--- of independencies. A superior example is medical epidemiology where the goal is to find the causes of a disease and exclude factors which are irrelevant. Whereas complete independence between two variables in a domain might be rare in reality (which would mean that the joint probability density of variables A and B can be factored:  p(A; B) = p(A)p(B)), conditional independence is more common and is often a result from true or apparent causality: consider the case that A is the cause of B  and B is the cause of C, then p(CjA; B)...
Improved Gaussian Mixture Density Estimates Using Bayesian Penalty Terms and Network Averaging.
We compare two regularization methods which can be used to improve  the generalization capabilities of Gaussian mixture density  estimates. The first method uses a Bayesian prior on the parameter  space. We derive EM (Expectation Maximization) update rules  which maximize the a posterior parameter probability. In the second  approach we apply ensemble averaging to density estimation.
Die besonderen Eigenschaften Neuronaler Netze bei der Approximation von Funktionen.

Efficient Methods for Dealing with Missing Data in Supervised Learning.
We present efficient algorithms for dealing with the problem of  missing inputs (incomplete feature vectors) during training and recall. Our approach is
Training Neural Networks with Deficient Data.
We analyze how data with uncertain or missing input features can be incorporated into the training of a neural network. The general solution requires a weighted integration over the unknown or uncertain input although computationally cheaper closed-form solutions can be found for certain Gaussian Basis Function (GBF) networks. We also discuss cases in which heuristical solutions such as substituting the mean of an unknown input can be harmful. 1 
Neuronale Netze zur Segmentierung und Clusterung von biomagnetischen Signalen.

Integrating Rule%2DBased Knowledge into Neural Computing.

Some Solutions to the Missing Feature Problem in Vision.
In visual processing the ability to deal with missing and noisy information is crucial. Occlusions and unreliable feature detectors often lead to situations where little or no direct information about features is available. However the available information is usually sufficient to highly constrain the outputs. We discuss Bayesian techniques for extracting class probabilities given partial data. The optimal solution involves integrating over the missing dimensions weighted by the local probability densities. The framework extends naturally to the case of noisy information. We show how to obtain closed-form approximations to the Bayesian solution using Gaussian basis function networks. Simulations on a complex task (3D hand gesture recognition) validate the theory. When both integration and weighting by input densities are used, performance decreases gracefully with the number of missing or noisy features. Performance is substantially degraded if either step is omitted. 1 
Network Structuring and Training Using Rule%2DBased Knowledge.
We demonstrate in this paper how certain forms of rule-based knowledge can be used to prestructure a neural network of normalized basis functions and give a probabilistic interpretation of the network architecture. We describe several ways to assure that rule-based knowledge is preserved during training and present a method for complexity reduction that tries to minimize the number of rules and the number of conjuncts. After training, the refined rules are extracted and analyzed.   Mail address: Siemens AG, Central Research, Otto-Hahn-Ring 6, 8000 Munchen 83.  1 INTRODUCTION  Training a network to model a high dimensional input/output mapping with only a small amount of training data is only possible if the underlying map is of low complexity: a more complex mapping requires a more complex network which results in high parameter variances and, as a consequence, in a high prediction error. This predicament can be solved if we manage to incorporate prior knowledge to bias the network as...
A Neural Architecture for 2D and 3D Vision.

Neural Control for Rolling Mills Incorporating Domain Theories to Overcome Data Deficiency.
In a Bayesian framework, we give a principled account of how domainspecific prior knowledge such as imperfect analytic domain theories can be optimally incorporated into networks of locally-tuned units: by choosing a specific architecture and by applying a specific training regimen. Our method proved successful in overcoming the data deficiency problem in a large-scale application to devise a neural control for a hot line rolling mill. It achieves in this application significantly higher accuracy than optimally-tuned standard algorithms such as sigmoidal backpropagation, and outperforms the state-of-the-art solution. 1 INTRODUCTION  Learning in connectionist networks typically requires many training examples and relies more or less explicitly on some kind of syntactic preference bias such as &#034;minimal architecture&#034; (Rumelhart, 1988; Le Cun et al., 1990; Weigend, 1991; inter alia) or a smoothness constraint operator (Poggio et al., 1990), but does not make use of explicit representations...
A Neural Network Approach for Three%2DDimensional Object Recognition.

